System and method for camera-based heart rate tracking

ABSTRACT

A system and method for camera-based heart rate tracking. The method includes: determining bit values from a set of bitplanes in a captured image sequence that represent the HC changes; determining a facial blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes; applying a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals; applying a Hilbert transform to each of the blood flow data signals; adjusting the blood flow data signals from revolving phase-angles into linear phase segments; determining an instantaneous heart rate for each the blood flow data signals; applying a weighting to each of the instantaneous heart rates; and averaging the weighted instantaneous heart rates.

TECHNICAL FIELD

The following relates generally to detection of a human heartbeat andmore specifically to a system and method for camera-based heart ratetracking.

BACKGROUND

The human heartbeat, or cardiac cycle, represents one of the primaryvital signs monitored by health care providers and members of thegeneral public alike. Heartbeat, as used herein, refers to a completeheartbeat, or a set of heartbeats, from its generation to the beginningof the next beat; thus, it includes the diastole, the systole, and theintervening pause. The pace of the heartbeats, referred to herein as theheart rate, is a measure of cardiac cycles per time period. Heart rateis typically measured in beats-per-minute (BPM) as a measure of, onaverage, how many cardiac cycles occur per minute. The BPM measurementcan be an average heart rate, measuring the average BPM over a sizeableperiod of cardiac cycles, or an instantaneous heart rate, measuring theBPM over a short period of cardiac cycles and extrapolating out the BPM.

Conventionally, the heart rate is measured using equipment such as anelectrocardiogram by recording the electrical activity of the heart overa period of time using electrodes placed on the skin. This approach is asignificant expense and requires invasive electrodes to be placed on asubject. Other conventional approaches include attaching a heart ratemonitor to a subject, which typically includes a chest strap transmitterand a receiver. This approach is not particularly accurate andsusceptible to noise, and in addition, requires the subject to place thetransmitter under his/her clothes. Further types of strapless heart ratemonitors allow the measurement of the heart rate with a wearable device,such as a wristwatch or finger clasp, by utilising an infrared sensor tomeasure the heart rate. However, such devices do not provide much detailand are not particularly accurate.

SUMMARY

In an aspect, there is provided a method for camera-based heart ratetracking of a human subject, the method comprising: receiving a capturedimage sequence of light re-emitted from the skin of the human subject;determining, using a machine learning model trained with a hemoglobinconcentration (HC) changes training set, bit values from a set ofbitplanes in the captured image sequence that represent the HC changesof the subject, the set of bitplanes being those that are determined toapproximately maximize a signal-to-noise ratio (SNR), the HC changestraining set comprising bit values from each bitplane of images capturedfrom a set of subjects for which heart rate is known; determining afacial blood flow data signal for each of a plurality of predeterminedregions of interest (ROIs) of the subject captured by the images basedon the HC changes; applying a band-pass filter of a passbandapproximating the heart rate to each of the blood flow data signals;applying a Hilbert transform to each of the blood flow data signals;adjusting the blood flow data signals from revolving phase-angles intolinear phase segments; determining an instantaneous heart rate for eachthe blood flow data signals; applying a weighting to each of theinstantaneous heart rates; averaging the weighted instantaneous heartrates; and outputting the average heart rate.

In a particular case, the ROIs are captured from the face of thesubject.

In another case, the ROIs are captured from the wrist or the neck of thesubject.

In yet another case, the ROIs are non-overlapping.

In yet another case, determining a set of bitplanes that maximize SNRcomprises: performing pixelwise image subtraction and addition ofbitplane vectors to maximize signal differences in all ROIs over apredetermined time period, and identifying bit values from bitplanesthat increase the signal differentiation and bit values from bitplanesthat decrease the signal differentiation or do not contribute to signaldifferentiation; and discarding the bit values from the bitplanes thatdecrease the signal differentiation or do not contribute to signaldifferentiation.

In yet another case, the machine learning model comprises a Long ShortTerm Memory (LSTM) neural network or a non-linear Support VectorMachine.

In yet another case, the passband is in a range of approximately 0.6hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1hertz.

In yet another case, determining the instantaneous heart rate for eachthe blood flow data signals comprises applying a differential filter tothe linear phase segments to convert the phase-angle data into frequencyunits representing a count value, the count value for each of the ROIsrepresents the instantaneous heart rate.

In yet another case, the method further comprising linearizing anddifferentiating the revolving phase-angles on a phase continuum scale todetermine the instantaneous heart rate.

In yet another case, the weighting is integrated over an interval in therange of approximately one second to ten seconds.

In yet another case, the weighting is integrated over an interval ofapproximately five seconds.

In another aspect, there is provided a system for camera-based heartrate tracking of a human subject, the system comprising one or moreprocessors and a data storage device, the one or more processorsconfigured to execute: a TOI module to receive a captured image sequenceof light re-emitted from the skin of a human subject, the TOI moduledetermines, using a machine learning model trained with a hemoglobinconcentration (HC) changes training set, bit values from a set ofbitplanes in the captured image sequence that represent the HC changesof the subject, the set of bitplanes being those that are determined toapproximately maximize a signal-to-noise ratio (SNR), the HC changestraining set comprising bit values from each bitplane of images capturedfrom a set of subjects for which heart rate is known, the TOI moduledetermines a facial blood flow data signal for each of a plurality ofpredetermined regions of interest (ROIs) of the subject captured by theimages based on the HC changes; a filtering module to apply a band-passfilter of a passband approximating the heart rate to each of the bloodflow data signals; a Hilbert transform module to apply a Hilberttransform to each of the blood flow data signals; an adjustment moduleto adjust the blood flow data signals from revolving phase-angles intolinear phase segments; a derivative module to determine an instantaneousheart rate for each the blood flow data signals; a weighting module toapply a weighting to each of the instantaneous heart rates; a summationmodule to average the weighted instantaneous heart rates; and an outputmodule to output the average heart rate.

In a particular case, the ROIs are captured from the face of thesubject.

In another case, the ROIs are non-overlapping.

In yet another case, the TOI module determines a set of bitplanes thatmaximize SNR by: performing pixelwise image subtraction and addition ofbitplane vectors to maximize signal differences in all ROIs over apredetermined time period, and identifying bit values from bitplanesthat increase the signal differentiation and bit values from bitplanesthat decrease the signal differentiation or do not contribute to signaldifferentiation; and discarding the bit values from the bitplanes thatdecrease the signal differentiation or do not contribute to signaldifferentiation.

In yet another case, the passband is in a range of approximately 0.6hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1hertz.

In yet another case, the derivative module determines the instantaneousheart rate for each the blood flow data signals by applying adifferential filter to the linear phase segments to convert thephase-angle data into frequency units representing a count value, thecount value for each of the ROIs represents the instantaneous heartrate.

In yet another case, the derivative module linearizes and differentiatesthe revolving phase-angles on a phase continuum scale to determine theinstantaneous heart rate.

In yet another case, the weighting applied by the weighting module isintegrated over an interval in the range of approximately one second toten seconds.

In yet another case, the weighting applied by the weighting module isintegrated over an interval of approximately five seconds.

These and other aspects are contemplated and described herein. It willbe appreciated that the foregoing summary sets out representativeaspects of camera-based heart rate tracking systems and methods for thedetermination of heart rate to assist skilled readers in understandingthe following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The features of the invention will become more apparent in the followingdetailed description in which reference is made to the appended drawingswherein:

FIG. 1 is an block diagram of a system for camera-based heart ratetracking, according to an embodiment;

FIG. 2 is a flowchart for a method for camera-based heart rate tracking,according to an embodiment;

FIG. 3 illustrates re-emission of light from skin epidermal andsubdermal layers;

FIG. 4 is a set of surface and corresponding transdermal imagesillustrating change in hemoglobin concentration for a particular humansubject at a particular point in time; and

FIG. 5 is a diagrammatic representation of a memory cell.

DETAILED DESCRIPTION

Embodiments will now be described with reference to the figures. Forsimplicity and clarity of illustration, where considered appropriate,reference numerals may be repeated among the Figures to indicatecorresponding or analogous elements. In addition, numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments described herein. However, it will be understood bythose of ordinary skill in the art that the embodiments described hereinmay be practiced without these specific details. In other instances,well-known methods, procedures and components have not been described indetail so as not to obscure the embodiments described herein. Also, thedescription is not to be considered as limiting the scope of theembodiments described herein.

Various terms used throughout the present description may be read andunderstood as follows, unless the context indicates otherwise: “or” asused throughout is inclusive, as though written “and/or”; singulararticles and pronouns as used throughout include their plural forms, andvice versa; similarly, gendered pronouns include their counterpartpronouns so that pronouns should not be understood as limiting anythingdescribed herein to use, implementation, performance, etc. by a singlegender; “exemplary” should be understood as “illustrative” or“exemplifying” and not necessarily as “preferred” over otherembodiments. Further definitions for terms may be set out herein; thesemay apply to prior and subsequent instances of those terms, as will beunderstood from a reading of the present description.

Any module, unit, component, server, computer, terminal, engine ordevice exemplified herein that executes instructions may include orotherwise have access to computer readable media such as storage media,computer storage media, or data storage devices (removable and/ornon-removable) such as, for example, magnetic disks, optical disks, ortape. Computer storage media may include volatile and non-volatile,removable and non-removable media implemented in any method ortechnology for storage of information, such as computer readableinstructions, data structures, program modules, or other data. Examplesof computer storage media include RAM, ROM, EEPROM, flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed by anapplication, module, or both. Any such computer storage media may bepart of the device or accessible or connectable thereto. Further, unlessthe context clearly indicates otherwise, any processor or controller setout herein may be implemented as a singular processor or as a pluralityof processors. The plurality of processors may be arrayed ordistributed, and any processing function referred to herein may becarried out by one or by a plurality of processors, even though a singleprocessor may be exemplified. Any method, application or module hereindescribed may be implemented using computer readable/executableinstructions that may be stored or otherwise held by such computerreadable media and executed by the one or more processors.

The following relates generally to detection of a human heartbeat andmore specifically to a system and method for camera-based heart ratetracking. Heart rate is determined using image processing techniquesperformed over a plurality of images captured by one or more digitalimaging cameras.

In embodiments of the system and method described herein, technicalapproaches are provided to solve the technological problem of detectingand tracking a human's heartbeat. The technical approaches describedherein offer the substantial advantages of both ‘spatial’ diversity,where region of interest (ROI) signals are acquired from non-overlappingdifferentially located regions on a human's face, and ‘time’ diversity,where accumulation of time-series data is simultaneously sampled withsynchronous or fixed timing. Applicant recognized the significantadvantages of this approach, for example, being that the quality of thebeats-per-minute (BPM) estimate is more robust to noise interference(for example due to outlier data) while retaining the ability to updatethe output BPM value at every sample interval (for example at the videoframe rate).

Applicant further recognized the significant advantages of the technicalapproaches described herein, for example, by utilizing machine learningtechniques, the composition of bitplanes of video images can beoptimized to maximize the signal to noise ratio of the heart rate band,especially as compared to conventional approaches.

Referring now to FIG. 1, a system for camera-based heart rate tracking100 is shown. The system 100 includes a processing unit 108, one or morevideo-cameras 105, a storage device 101, and an output device 102. Theprocessing unit 108 may be communicatively linked to the storage device101 which may be preloaded and/or periodically loaded with video imagingdata obtained from one or more video-cameras 105. The processing unit108 includes various interconnected elements and modules, including aTOI module 110, a filtering module 112, a Hilbert transform module 114,an adjustment module 116, a derivative module 118, a weighting module120, a summation module 122, and an output module 124. The TOI moduleincludes an image processing unit 104 and a filter 106. The video imagescaptured by the video-camera 105 can be processed by the filter 106 andstored on the storage device 101. In further embodiments, one or more ofthe modules can be executed on separate processing units or devices,including the video-camera 105 or output device 102. In furtherembodiments, some of the features of the modules may be combined or runon other modules as required.

The term “video”, as used herein, can include sets of still images.Thus, “video camera” can include a camera that captures a sequence ofstill images.

Using transdermal optical imaging (TOI), the TOI module 110 can isolatehemoglobin concentration (HC) from raw images taken from a traditionaldigital camera. Referring now to FIG. 3, a diagram illustrating there-emission of light from skin is shown. Light 301 travels beneath theskin 302, and re-emits 303 after travelling through different skintissues. The re-emitted light 303 may then be captured by opticalcameras 105. The dominant chromophores affecting the re-emitted lightare melanin and hemoglobin. Since melanin and hemoglobin have differentcolor signatures, it has been found that it is possible to obtain imagesmainly reflecting HC under the epidermis as shown in FIG. 4.

Using transdermal optical imaging (TOI), the TOI module 110, via theimage processing unit 104, obtains each captured image or video stream,from the camera 105, and performs operations upon the image to generatea corresponding optimized hemoglobin concentration (HC) image of thesubject. From the HC data, the facial blood flow localized volumeconcentrations can be determined; whereby localized volumeconcentrations refer to measured HC intensity values within a region ofinterest. As described, regions of interest are used to define alocalized bounded area, or areas, for which HC is to be measured Theimage processing unit 104 isolates HC in the captured video sequence. Inan exemplary embodiment, the images of the subject's faces are taken at30 frames per second using a digital camera 105. It will be appreciatedthat this process may be performed with alternative digital cameras,lighting conditions, and frame rates.

Isolating HC is accomplished by analyzing bitplanes in the videosequence to determine and isolate a set of the bitplanes thatapproximately maximize the signal to noise ratio (SNR). Thedetermination of high SNR bitplanes is made with reference to an HCtraining set of images constituting the captured video sequence, in somecases, supplied along with EKG, pneumatic respiration, blood pressure,laser Doppler data collected from the human subjects from which thetraining set is obtained.

The regions of interest (ROIs) of the human subject's face, for exampleforehead, nose, and cheeks, may be defined as stationary or dynamicallyupdated using the video images. The ROIs are preferably non-overlapping.These ROIs are preferably selected on the basis of knowledge in the artin respect of ROIs for which HC is particularly indicative of heart ratetracking (for example, forehead, cheek, or the like). Using nativeimages that consist of all bitplanes (typically 24 bitplanes for eachcolor image), signals that change over a particular time period (forexample, 10 seconds) on each of the ROIs are extracted. In some cases,the dynamically updated ROIs can be chosen and/or maintained by usingface-tracking software.

Bitplanes are a fundamental aspect of digital images. Typically, adigital image consists of certain number of pixels (for example, awidth×height of 1920×1080 pixels). Each pixel of the digital imagehaving one or more channels (for example, color channels red, green, andblue (RGB)). Each channel having a dynamic range, typically 8 bits perpixel per channel, but occasionally 10 bits per pixel per channel forhigh dynamic range images. Whereby, an array of such bits makes up whatis known as the bitplane. In an example, for each image of color videos,there can be three channels (for example, red, green, and blue (RGB))with 8 bits per channel. Thus, for each pixel of a color image, thereare typically 24 layers with 1 bit per layer. A bitplane in such a caseis a view of a single 1-bit map of a particular layer of the imageacross all pixels. For this type of color image, there are thereforetypically 24 bitplanes (i.e., a 1-bit image per plane). Hence, for a1-second color video with 30 frames per second, there are at least 720(30×24) bitplanes. In the embodiments described herein, Applicantrecognized the advantages of using bit values for the bitplanes ratherthan using, for example, merely the averaged values for each channel.Thus, a greater level of accuracy can be achieved for making predictionsof HC changes, and as described making predictions of heart rate,because employing bitplanes provides a greater data basis for trainingthe machine learning model.

The raw signals can be pre-processed using one or more filters,depending on the signal characteristics. Such filters may include, forexample, a Butterworth filter, a Chebycheff filter, or the like. Usingthe filtered signals from two or more ROIs, machine learning is employedto systematically identify bitplanes that will significantly increasethe signal differentiation (for example, where the SNR improvement isgreater than 0.1 db) and bitplanes that will contribute nothing ordecrease the signal differentiation. After discarding the latter, theremaining bitplane images can optimally determine the bold flow.

The machine learning process involves manipulating the bitplane vectors(for example, 24 bitplanes×60 hz) using the bit value in each pixel ofeach bitplane along the temporal dimension. In one embodiment, thisprocess requires subtraction and addition of each bitplane to maximizethe signal differences in all ROIs over the time period. In some cases,to obtain reliable and robust computational models, the entire datasetcan be divided into three sets: the training set (for example, 80% ofthe whole subject data), the test set (for example, 10% of the wholesubject data), and the external validation set (for example, 10% of thewhole subject data). The time period can vary depending on the length ofthe raw data (for example, 15 seconds, 60 seconds, or 120 seconds). Theaddition or subtraction is performed in a pixel-wise manner. An existingmachine learning algorithm, the Long Short Term Memory (LSTM) neuralnetwork, or a suitable alternative thereto is used to efficiently andobtain information about the improvement of differentiation in terms ofaccuracy, which bitplane(s) contributes the best information, and whichdoes not in terms of feature selection. The Long Short Term Memory(LSTM) neural network allow us to perform group feature selections andclassifications. The LSTM machine learning algorithm are discussed inmore detail below. From this process, the set of bitplanes to beisolated from image sequences to reflect temporal changes in HC isobtained. An image filter is configured to isolate the identifiedbitplanes as described below.

To extract facial blood flow data, facial HC change data on each pixelof each subject's face image is extracted as a function of time when thesubject is being viewed by the camera 105. To increase signal-to-noiseratio (SNR), the subject's face is divided into a plurality of regionsof interest (ROIs) according to, for example, their differentialunderlying physiology, and the data in each ROI is averaged.

Machine learning approaches (such as a Long Short Term Memory (LSTM)neural network, or a suitable alternative such as non-linear SupportVector Machine) and deep learning may be used to assess the existence ofcommon spatial-temporal patterns of hemoglobin changes across subjects(for example, differences in amplitude in blood flow changes in theforehead and the cheek over time). In some cases, the Long Short TermMemory (LSTM) neural network, or an alternative, can be trained on thetransdermal data from a portion of the subjects (for example, 80%, or90% of the subjects) to obtain a computational model for the facialblood flow, which can be tested using the test data set and externallyvalidated using the external validation data set.

Once the model is trained as described, it becomes possible to obtain avideo sequence of any subject and apply the HC extracted from selectedbitplanes to the computational models to determine blood flow. For longrunning video streams with changes in blood flow and intensityfluctuations, changes of the estimation and intensity scores over timerelying on HC data based on a moving time window (e.g., 10 seconds) maybe reported.

In an example using the Long Short Term Memory (LSTM) neural network,the LSTM neural network comprises at least three layers of cells. Thefirst layer is an input layer, which accepts the input data. The second(and perhaps additional) layer is a hidden layer, which is composed ofmemory cells (see FIG. 5). The final layer is output layer, whichgenerates the output value based on the hidden layer using LogisticRegression.

Each memory cell, as illustrated, comprises four main elements: an inputgate, a neuron with a self-recurrent connection (a connection toitself), a forget gate and an output gate. The self-recurrent connectionhas a weight of 1.0 and ensures that, barring any outside interference,the state of a memory cell can remain constant from one time step toanother. The gates serve to modulate the interactions between the memorycell itself and its environment. The input gate permits or prevents anincoming signal to alter the state of the memory cell. On the otherhand, the output gate can permit or prevent the state of the memory cellto have an effect on other neurons. Finally, the forget gate canmodulate the memory cell's self-recurrent connection, permitting thecell to remember or forget its previous state, as needed.

The equations below describe how a layer of memory cells is updated atevery time step t. In these equations:

x_(t) is the input array to the memory cell layer at time t. In ourapplication, this is the blood flow signal at all ROIs{right arrow over (x)} _(t)=[x _(1t) ,x _(2t) . . . x _(nt)]

-   -   W_(i), W_(f), W_(c), W_(o), U_(i), U_(f), U_(c), U_(o) and V_(o)        are weight matrices; and    -   b_(i), b_(f), b_(c) and b_(o) are bias vectors

First, we compute the values for i_(t), the input gate, and {tilde over(C)}_(i) the candidate value

for the states of the memory cells at time t:i _(t)=σ(W _(i) x _(i) +U _(i) h _(t-1) +b _(i)){tilde over (C)} _(t)=tan h(W _(c) x _(t) +U _(c) h _(t-1) +b _(c))

Second, we compute the value for f_(t), the activation of the memorycells' forget gates at time t:f _(t)=σ(W _(f) x _(t) +U _(f) h _(t-1) +b _(f))

Given the value of the input gate activation i_(t), the forget gateactivation f_(t) and the candidate state value {tilde over (C)}_(t), wecan compute C_(t) the memory cells' new state at time t:C _(t) =i _(t) *{tilde over (C)} _(t) +f _(t) *C _(t-1)

With the new state of the memory cells, we can compute the value oftheir output gates and, subsequently, their outputs:o _(t)=σ(W _(o) x _(t) +U _(o) h _(t-1) +V _(o) C _(t) +b _(o))h _(t) =o _(t)*tan h(C _(t))

Based on the model of memory cells, for the blood flow distribution ateach time step, we can calculate the output from memory cells. Thus,from an input sequence x₀, x₁, x₂, . . . , x_(n), the memory cells inthe LSTM layer will produce a representation sequence h₀, h₁, h₂, . . ., h_(n).

The goal is to classify the sequence into different conditions. TheLogistic Regression output layer generates the probability of eachcondition based on the representation sequence from the LSTM hiddenlayer. The vector of the probabilities at time step t can be calculatedby:p _(t)=softmax(W _(output) h _(t) +b _(output))where w_(output) is the weight matrix from the hidden layer to theoutput layer, and b_(output) is the bias vector of the output layer. Thecondition with the maximum accumulated probability will be the predictedcondition of this sequence.

The heart rate tracking approach, used by the system 100 on the HCchange data from the TOI module 110, utilizes adaptive weighting ofmultiple regions-of-interest (ROIs), and uses minimizing ‘noise’criteria to control the weights. The heart rate tracking approach alsoutilizes a Hilbert transform to extract a coherent signal for theheartbeat. Advantageously, the accuracy when measured against ‘groundtruth’ electrocardiogram (ECG) data indicates that the estimated“beats-per-minute” (BPM) of the heartbeat recovery approach to betypically consistent within +/−2 BPM of the ECG data.

The blood flow localized volume concentrations data captured by the TOImodule 110, as described herein, of a human subject's face, as either‘live’ or previously recorded, is used as the source data fordetermining the subject's heart rate. The facial blood flow data canthen be used for estimation of related parameters such as the averageheart rate in BPM.

The blood flow data signal is specified by the interpretation of the HCchanges. As an example, the system 100 can monitor stationary HC changescontained by a selected ROI over time, by observing (or graphing) theresulting temporal profile (for example, shape) of the selected ROI HCintensity values over time. In some cases, the system 100 can monitormore complex migrating HC changes across multiple ROIs by observing (orgraphing) the spatial dispersion (HC distribution between ROIs) as itevolves over time.

In order to estimate the BPM of the human subject, the TOI module 110detects, recovers and tracks the valid occurrences of the subject'sheartbeat. The system 100 through its various modules, as describedherein, then converts these periodic occurrences into an instantaneousstatistic representing the average count as BPM. This instantaneousstatistic is then continuously updated. Advantageously, this approachhas data-sampling that is equal to the video acquisition frame-ratespecified as “frames-per-second” (FPS). This provides a continuousper-frame estimation of the instantaneous heart rate.

Turning to FIG. 2, a flowchart for a method for camera-based heart ratetracking 200 is shown.

At block 202, facial blood flow is extracted from the video usingtransdermal optical imaging by the TOI module 110, as described herein,for localized volume concentrations at defined regions-of-interest (ROI)on the face. In addition, the TOI module 110 records dynamic changes ofsuch localized volume concentrations over time.

At block 204, the blood flow volume concentrations data from each ROIare treated by the filtering module 112 as an independent signal. Thus,the blood flow data for each ROI is routed through a separate,individual corresponding signal processing path (also known as chain)which handles the specific TOI signal originating from a unique locationon the facial image. In this way, multiple ROIs are generating multiplesignals which are independently yet concurrently processed, as a bank ofROI signal chains, using the digital signal processing (DSP) techniquesdescribed herein.

In an example, the face can be divided into 17 different regions ofinterest according to facial anatomy or the underlying distributions offacial vasculature (for example, the nose, the forehead, and the like).In this case, there will be 17 separate ROI signal chains, eachprocessing a unique signal extracted from the facial image. The groupingof these 17 ROI signal chains is collectively referred to as a bank ofROI chains. As will be described, the signal processing of each ROIsignal chain can be identical across all the ROIs, such that the sameoperations are concurrently being applied to each separate ROI signalpath.

The dimension spanning across multiple ROIs will be referred to hereinas a spatial diversity axis of the ROI signal banks. Each ROI signalchain includes an incoming stream of images, such as from a videocamera, separated by an interval period (as described herein). Thedimension spanning across images for each of the ROI signal chains,along the time dimension, will be referred to herein as the timediversity axis.

At block 206, the filtering module 112 routes each of the ROI blood flowsignals to its corresponding position in a bank of digitalband-pass-filters (BPF) for processing. The passband for these filtersis chosen to cover the extended frequency range representing theheart-rate (where 60 bpm=1 bps=1 hz). This filtering of the signal isrequired to reduce energy content outside of a period of the heart-rateand thereby improving the signal-to-noise ratio (SNR). In an example, aninitial heart-band passband range can extend between 0.6 hertz to 1.2hertz. Although each individual ROI signal is filtering the heart beatfrom a spatially unique location on the face, the subject heart beat canbe a global signal. Therefore, in some cases, a common subject-specificperiod can be observed across all ROIs of the subject. Thus, in somecases, the active passband for all ROIs can also be dynamically andadaptively adjusted to a common range.

Each of the filtered ROI signals, represented as a time-series, are thenreceived, at block 208, by the Hilbert transform module 114. The Hilberttransform module 114 applies a Hilbert transform (HT) to the filteredsignal. Each ROI signal is thus converted to its analytic (complex)equivalent signal attributes and decomposed as both instantaneousamplitude and instantaneous phase.

At block 210, the instantaneous phase components for each ROI signal inthe signal bank are adjusted, by the adjustment module 116, fromrevolving phase-angles into linear phase segments in order to resolveabsolute timing differences. Since the sampling steps are constantintervals, for example at the video frame rate, the rate of changebetween discrete instantaneous phase steps can represent a frequency. Inthis case, the frequency is equivalent to an integer count of theheartbeat events (occurrences) over the specified interval. To determinethe rate of change between discrete instantaneous phase steps, at block212, the instantaneous phase profile for each ROI signal is routed tothe derivative module 118, which applies a differential filter, toconvert the phase-angle information into frequency units (also calledevent units), which represent a statistic count value. This count valueper ROI reflects the instantaneous BPM estimate as a continuous signal.

In this case, due to the captured sampling data coming from a stream ofvideo images with a consistent frame-rate, accurate phase-angles can bedetermined based on a known timing reference, which in this case is theframes-per-second. The phase angles can then be linearized on a phasecontinuum scale, and the phase steps can be differentiated on the phasecontinuum scale to determine the frequency. This frequency iseffectively the rate of heartbeat occurrences, also known as the heartrate. For proper determination of the heart rate, the sampling rateneeds to have finer granularity than the measured quantity, the heartrate. In this case, processing at the video frame-rate (fps) satisfiesthis condition.

Phase angles can be linearized (or compensated) through a process knownas “unwrapping” or “unfolding” the continuously overlapping range ofphase angle response (0 to 2*pi radians). This linearization processensures the correct accumulation of the “rotating” phase angles whenevernormalizing the total phase delay which may exceed one period (2*pi) ofthe signal frequency. After this normalization all phase delays fromvarious ROIs may be directly compared against each other

At block 214, the weighting module 120 then applies a weighting to eachof the differentially filtered signals. In a particular case, theweighting module 120 applies the following weighting to each of thedifferentially filtered ROI signals: W(i)=1/(STD (dP)){circumflex over( )}2 integrated over a 5 second interval. Whereby, ‘STD’ is astatistical standard-deviation function measurement, ‘dP’ is the phasedelta over the interval ‘i’, and W(i) is the resulting weightcoefficient. The weighting represents an inverse relationship betweennoise, which is modelled as exhibiting randomized, uncoherent qualitiesand having a high standard deviation, and the differentially filteredheart rate signal, which is slowly changing but coherent. The weightingmodule 120 then applies a moving window to this weighting to update eachof the ROI signals weighting for the specific interval. The contributionof the signal, representing the BPM estimate, from individual ROI signalbanks will each be scaled by the respective weighting output. Thescaling will be inversely proportional to the magnitude of each signal'scalculated weights. In further cases, a different interval may be used,for example, 1 second, 2, second, 10 second, or the like.

All ROI signal banks will terminate their respective output signals,representing the instantaneous BPM estimate, at the summation module122. At block 216, the summation module 122 will determine the averageBPM based on the adaptively scaled contributions from all the ROIs. Atblock 218, the output module 124 will then output the calculated averageBPM to an output device; for example, to a computer monitor, an LCDscreen on a wearable device, or the like.

Applicant recognized the substantial advantages of using amulti-dimensional approach, as described herein, which offers thebenefits of both ‘spatial’ diversity and ‘time’ diversity. Spatialdiversity allows ROI signals to be acquired from non-overlappingdifferentially located regions on the human subject's face. ‘Time’diversity allows accumulation of time-series data which issimultaneously sampled with a synchronous or fixed timing. Applicantrecognized that a significant advantage of this approach being that thequality of the BPM estimate is more robust to noise interference (forexample outlier data), and therefore more accurate than conventionalapproaches, while retaining the ability to update the output BPM valueat every sample interval (in this example, at the video frame rate).

As an example, outlier data can distort the HC determinations and dueto, for example, uneven lighting conditions on the face, slowly changingshadows moving across the face, or fixed facial obfuscations such aswrinkles, glasses, hair, and the like. With the multi-dimensionalapproach, as described herein, leveraging the spatial dimension bymeasuring the same signal at different points on the subject's face, thesystem is able to reject inconsistent or outlier data. As an example,having the ROI signal chains capturing approximately the same globalheart-beat signal from 17 different points on the subject's face. Insome cases, an average of the 17 ROI signals, with equal weighting, mayreduce some outlier effects. As a further refinement, and for furtheraccuracy, the multi-dimensional approach, as described herein, applies aweighted average to determine heart rate, whereby the weights beingadaptively calculated to minimize data which has higher volatility.

In further embodiments, the system 100 could use an asynchronous samplerate. The asynchronous sample rate can capture HC data from images at arate not synchronized or coupled with the video frame-rate. For example,capture the HC data at approximately 1 hertz, meaning 1 beat-per-secondor 60 BPM nominal rate. Then, according to the Nyquist sampling theory,sampling at a minimum of twice the highest signal rate. For example,sampling at 5 hertz (or 5 frames per second), which would be much higherthan required. In addition, this sampling would have the benefit ofallowing the system 100 to only have to process 5 frames-per-second,rather than the more computationally intensive rates such as 30 fps or60 fps.

In further embodiments, the camera can be directed to the skin ofdifferent body parts, such as for example the wrist or neck. From thesebody areas, the system may also extract dynamic hemoglobin changes todetermine blood flow, and thus acquire heart rate as described herein.In some cases, optical sensors pointing, or directly attached to theskin of any body parts such as for example the wrist or forehead, in theform of a wrist watch, wrist band, hand band, clothing, footwear,glasses or steering wheel may be used. From these body areas, the systemmay also extract blood flow data for heart rate determinations.

In still further embodiments, the system may be installed in robots andtheir variables (e.g., androids, humanoids) that interact with humans toenable the robots to track heart rate on the face or other-body parts ofhumans whom the robots are interacting with.

The foregoing system and method may be applied to a plurality of fields.In one embodiment the system may be installed in a smartphone device toallow a user of the smartphone to measure their heart rate. In anotherembodiment, the system may be provided in a video camera located in ahospital room to allow the hospital staff to monitor the heart rate of apatient without requiring invasive monitors.

Further embodiments can be used in police stations and border stationsto monitor the heart rate of suspects during interrogation. In yetfurther embodiments, the system can be used in marketing to see theheart rate changes of consumers when confronted with specific consumergoods.

Other applications may become apparent.

Although the invention has been described with reference to certainspecific embodiments, various modifications thereof will be apparent tothose skilled in the art without departing from the spirit and scope ofthe invention as outlined in the claims appended hereto. The entiredisclosures of all references recited above are incorporated herein byreference.

The invention claimed is:
 1. A method for camera-based heart ratetracking of a human subject, the method comprising: receiving a capturedimage sequence of light re-emitted from the skin of the human subject;determining, using a machine learning model trained with a hemoglobinconcentration (HC) changes training set, HC changes of the subject;determining a blood flow data signal for each of a plurality ofpredetermined regions of interest (ROIs) of the subject captured by theimages based on the HC changes; determining an instantaneous heart ratefor each the blood flow data signals; applying a weighting to each ofthe instantaneous heart rates; averaging the weighted instantaneousheart rates; and outputting the average heart rate.
 2. The method ofclaim 1, wherein the ROIs are captured from one or more of: the face,the wrist or the neck of the subject.
 3. The method of claim 1, whereinthe ROIs are non-overlapping.
 4. The method of claim 1, wherein the HCchanges of the subject are represented by bit values from a set ofbitplanes in the captured image sequence, the set of bitplanes beingthose that are determined to approximately maximize a signal-to-noiseratio (SNR), the HC changes training set comprising bit values from eachbitplane of images captured from a set of subjects for which heart rateis known.
 5. The method of claim 4, wherein determining the set ofbitplanes that maximize SNR comprises: performing pixelwise imagesubtraction and addition of bitplane vectors to maximize signaldifferences in all ROIs over a predetermined time period; identifyingbit values from bitplanes that increase the signal differentiation andbit values from bitplanes that decrease the signal differentiation or donot contribute to signal differentiation; and discarding the bit valuesfrom the bitplanes that decrease the signal differentiation or do notcontribute to signal differentiation.
 6. The method of claim 1, whereinthe machine learning model comprises a Long Short Term Memory (LSTM)neural network or a non-linear Support Vector Machine.
 7. The method ofclaim 1, wherein the method further comprises applying a band-passfilter of a passband approximating the heart rate to each of the bloodflow data signals, the passband being in a range of approximately 0.6hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1hertz.
 8. The method of claim 1, wherein determining the instantaneousheart rate for each the blood flow data signals comprises adjusting theblood flow data signals from revolving phase-angles into linear phasesegments, and applying a differential filter to the linear phasesegments to convert phase-angle data into frequency units representing acount value, the count value for each of the ROIs represents theinstantaneous heart rate.
 9. The method of claim 8, further comprisinglinearizing and differentiating the revolving phase-angles on a phasecontinuum scale to determine the instantaneous heart rate.
 10. Themethod of claim 1, wherein the weighting is integrated over an intervalin the range of approximately one second to ten seconds.
 11. A systemfor camera-based heart rate tracking of a human subject, the systemcomprising one or more processors and a data storage device, the one ormore processors configured to execute: a TOI module configured to:receive a captured image sequence of light re-emitted from the skin of ahuman subject; determine, using a machine learning model trained with ahemoglobin concentration (HC) changes training set, HC changes of thesubject; and determine a blood flow data signal for each of a pluralityof predetermined regions of interest (ROIs) of the subject captured bythe images based on the HC changes; a derivative module to determine aninstantaneous heart rate for each the blood flow data signals; aweighting module to apply a weighting to each of the instantaneous heartrates; a summation module to average the weighted instantaneous heartrates; and an output module to output the average heart rate.
 12. Thesystem of claim 11, wherein the ROIs are captured from one or more of:the face, the wrist or the neck of the subject.
 13. The system of claim11, wherein the ROIs are non-overlapping.
 14. The system of claim 11,wherein the HC changes of the subject are represented by bit values froma set of bitplanes in the captured image sequence, the set of bitplanesbeing those that are determined to approximately maximize asignal-to-noise ratio (SNR), the HC changes training set comprising bitvalues from each bitplane of images captured from a set of subjects forwhich heart rate is known.
 15. The system of claim 14, wherein the TOImodule determines a set of bitplanes that maximize SNR by: performingpixelwise image subtraction and addition of bitplane vectors to maximizesignal differences in all ROIs over a predetermined time period;identifying bit values from bitplanes that increase the signaldifferentiation and bit values from bitplanes that decrease the signaldifferentiation or do not contribute to signal differentiation; anddiscarding the bit values from the bitplanes that decrease the signaldifferentiation or do not contribute to signal differentiation.
 16. Thesystem of claim 11 further comprising a band-pass filter having apassband approximating the heart rate applied to each of the blood flowdata signals, wherein the passband is in a range of approximately 0.6hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1hertz.
 17. The system of claim 11, wherein the derivative moduledetermines the instantaneous heart rate for each the blood flow datasignals by adjusting the blood flow data signals from revolvingphase-angles into linear phase segments, and applying a differentialfilter to the linear phase segments to convert the phase-angle data intofrequency units representing a count value, the count value for each ofthe ROIs represents the instantaneous heart rate.
 18. The system ofclaim 17, wherein the derivative module linearizes and differentiatesthe revolving phase-angles on a phase continuum scale to determine theinstantaneous heart rate.
 19. The system of claim 11, wherein theweighting applied by the weighting module is integrated over an intervalin the range of approximately one second to ten seconds.
 20. The systemof claim 11, wherein the weighting applied by the weighting module isintegrated over an interval of approximately five seconds.